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Abstract 

This paper describes the evaluation of high quality learning designs which are being 
selected for possible redevelopment in a National Project funded by the Australian 
University Teaching Committee (AUTC). The project focuses on “Information and 
Communication Technologies (ICTs) and Their Role in Flexible Learning” and is 
evaluating over 50 projects with a view to developing a range of software tools, 
templates and/or guidelines based on those that are de emed to be effective ICT - 
based learning projects. The approach is unique in that it tries to pinpoint the key 
attributes of ICT-based projects that make them suitable for application in other 
contexts and in other knowledge domains. 
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Introduction 

There tends to be general consensus among experts that the forms of learning environments most effective for 
meaningful learning in higher education are those that are based on the contemporary theories of learning which 
support knowledge construction through learner -centred settings (e.g. Duffy & Cunningham, 1996; Bostock, 
1998). These perspectives about learning are challenging conventional teaching approaches. For example, 
Cunningham et al. (1998) state: 

The growing acceptance of new educational philosophies and practices, such as constructivism 
and action learning during the 1980s, have challenged the valence of the didactic 
lecture/tutorial/textbook model common in higher education, promoted the notions of the 
academic role as ‘a guide on the side’ rather than ‘the sage on stage’, and conceived of the student 
role as one of independent self-directed learner, (p. 25) 

The growing awareness of effective and meaningful teaching and learning plus the recent developments in ICT 
has led to synergies emerging between the use of ICT and the adoption of powerful learning strategies. The Web 
is one technology that shows particular promise for supporting meaningful learning through its remarkable 
functionality, support for flexible delivery modes and capacity to link and connect those involved in the learning 
process (e.g. Levin, 1999). The possibilities exist for rich learning based on this technology, but for the most 
part, pedagogically sound and exciting Web courseware tools have yet to be developed to take advantage of such 
opportunities. 

One of the key issues is that the pace of change of emerging Web technologies is so rapid that pedagogical 
models may be needed to help create Web tools from a learner-centred perspective. Salomon (1998) has 
supported this concern and has noted that for the first time in history, technologies are outpacing pedagogical 
and psychological rationale. However, a body of literature is starting to report on innovative tools, with strong 
pedagogical underpinning. Bonk (1998) has reported on interactive tools for on-line portfolio feedback, profile 
commenting, and Web link rating. Oliver and McLoughlin (1999) are building tools for on-line debate, 
reflection, concept mapping and student surveying and discussion. Wills, Ip & Bunnett (2000) are building 
engines for online role plays. As a way of describing the range of options that might contribute to a learning 
design in such contexts Figure 1 shows the combination of elements that might be considered in such an 
endeavour. 

In this project, the terms “learning designs”, “high quality learning experiences”, and “flexible learning” are 
defined as follows: Learning designs: refer to a variety of designs that support student learning experiences. 
Learning designs may be at the level of a whole subject, subject component or learning resource. High quality 
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learning experiences : refer to experiences resulting from an environment, which encourages students to seek 
understanding rather than memorisation (only for the purposes of assessment), and which encourage the 
development of lifelong learning skills. Flexible learning, refers to an educational approach that meets the 
diverse needs of students. The project is to focus on how ICT can be used to design flexible opportunities for 
students. 




Figure 1: Elements of learning designs for online learning 

Current settings hold fewer impediments to ICT uptake than have been present in the past. Universities within 
Australia have moved swiftly in recent years to develop the necessary infrastructure to support ICT as a delivery 
medium and most universities now boast a solid ICT infrastructure aimed at supporting teaching and learning 
programs. The uptake of ICT as a delivery medium has been supported by professional development programs 
and activities aiming to develop the ICT literacy of staff plus clearinghouses and Web sites for dissemination of 
information about ICT in teaching. Funding has been applied by government sources to support the development 
of university teaching and learning and many organisations now exist that support and promote quality teaching 
as a scholarly pursuit. Among the major impediments that still stand are the lack of quality teaching and learning 
models and appropriate instructional material and software for teachers to apply. 

This project aims to provide some relief to these impediments by identifying and creating quality resources for 
generic and mainstream application and by providing appropriate support and resources that will guide and 
encourage their use. Projects of this type should result in a coming decade that witnesses a growth in 
pedagogically based learning technologies. 



Aims of this project 

The aim of this project is to assist university instructors to create high quality flexible learning experiences for 
students by providing a range of generic resources/tools/templates/guidelines that draw upon successful flexible 
learning projects that utilise ICT and which may be generalised beyond the scope of the individual project. 
Successful ICT -based learning projects are those that facilitate high quality learning experiences for students. A 
study conducted by Alexander and McKenzie (1998) highlighted that one contributing factor towards a 
successful learning outcome for an ICT -based learning project was the learning design employed. Thus, this 
project has followed the process: 

1. Identification of a range of learning designs that have been demonstrated to contribute to high 
quality learning experiences and which can be applied generically; 
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2. Design and subsequent development of a series of re-usable software, templates and/or guidelines 
for the learning designs previously identified; and 

3. Dissemination of good practice for the use of or implementation of the software, templates and/or 
guidelines in new contexts. 

Crucial to the success of this project was the development of an evaluation instrument referred to by the project 
as an Evaluation and Redevelopment Framework with a twofold purpose: 

1. To facilitate the identification of learning designs that foster high quality learning experiences; and 

2. to provide a mechanism to determine whether such learning activity designs have the potential for 
re-development in a more generic form. 



Project structure 

The project began in November 2000 and is structured against four milestones: 

1. Milestone One (May 2001): Development of the Evaluation and Redevelopment Framework 

2. Milestone Two (November 2001): Identification and documentation of learning designs that foster 
high quality learning experiences and that have the potential for redevelopment in a more generic 
form. 

3. Milestone Three (June 2002): Development of a selected number of learning designs in a generic 
form to at least prototype stage. 

4. Milestone Four (December 2002): Completion of the development of learning designs in a more 
generic form and finalisation of a web site that will store the project’s developed resources. 



Development of the Evaluation and Redevelopment Framework 

Characterising High Quality Learning 

A major project activity has been the critique of what constitutes “high quality learning”. Professor Boud and 
Associate Professor Prosser were commissioned, as two leading thinkers about learning in higher education in 
Australia, to develop a paper on high quality learning. Their ideas together with feedback from the project team 
led to the development of a set of “Key Principles for High Quality Student Learning in Higher Education — 
from a Learning Perspective” (Boud & Prosser, 2001). The key principles describe four main characteristics that 
underpin high quality learning in the higher education context. The principles are elaborated through a series of 
questions that provide a lens through which learning environments can be explored. The four principles are 
holistic in that they incorporate both learning outcomes and learning processes and are based upon an 
experience- based learner-centred view of learning. The four principles are outlined below in the form of 
descriptions of high quality learning activities. 

High quality learning activities: 

1. Engage learners through: 

■ Building on their learning intents generally and their particular expectations of the activity in question; 

■ Acknowledging and taking account of their prior experience, both their knowledge and experience of 
situations which might impinge on the present ones; 

■ Mobilising their will and desire and developing some kind of emotional engagement with the task in 
hand; 

■ Providing them with a sense of agency with respect to the activity or significant parts of it; and 

■ Recognising that learning is a social act and involves other learners for at least part of the activity. 

Z Acknowledge context through: 

■ Involvement with problems in context; 

■ Recognising the context of the learner (who may see themselves as decontexualised); 

■ Maintaining an awareness of the cultural assumptions and stereotyping which may be incorporated in 
the context; 

■ Situating learning tasks within disciplinary or professional or practical knowledge as appropriate; 

■ Taking account of the site of application of what is to be learned (this poses different challenges when 
the learner is currently engaged in the site of application and when they are not); 

■ Appreciating the knowledge demands on students and equipping them to deal with them; and 

■ Ensuring that there is a clear alignment between the activities in whidi students will be engaged and the 
ways in which they will be assessed. 



3. Challenge learners through: 

■ Prompting them to seek and discern variation in the knowledge and experiences in which they are 
involved; 

■ Questioning the assumptions they bring to the activity and the assumptions they develop through it; 

■ Encouraging them to see what is provided as a means to wider ends and go beyond what is provided; 
and 

■ Creating situations in which they are required to take responsibility for their own learning and to shape 
the activity to their own ends. 

4. Involve practice through: 

■ Demonstrating what has been learned for themselves and for others; 

■ Gaining feedback at strategic points in learning, but also recognising that finding ways of gaining 
feedback for one self other than that provided is also important; 

■ Reflecting on and making sense of their experiences. Continuous exposure to new activities without 
integration and consolidation within the learner’s framework is not conducive to good learning; and 

■ Developing confidence in performance from practice. 



Developing the evaluation instrumentation 

Whilst the above principles formed the basis of the evaluation framework, the following issues were raised by 
the reviewers that they should be incorporated into the instrument: 

■ How technology is embedded in a learning design and how its use supports or hinders the learning 
experience. 

■ The issues of scalability, transferability, and technology affordances. 

■ To determine suitability of redevelopment of a learning design, the evaluation should provide a 
mechanism to glean the critical design features from a learning design and consider how these design 
features could be implemented in a more generic form. 

■ To place the review framework within a staged process which might inform the project through a series 
of critical decision points. 

The first complete version of the evaluation instrument was devised by the Core Team and Research Team after 
the first workshop. This version was formatively evaluated in the second workshop (s cheduled one month after 
the first). The Research Team also examined existing evaluative instruments to determine whether these could 
inform and/or be incorporated into the project’s evaluation framework. 

Since the second workshop (held at the end of April 2001) the evaluation framework has undergone further 
review and formative evaluation. Feedback from the Project Review Panel and International Reference Group 
has been considered and via discussions with the Core Team, Research Team and Steering Committee, a revised 
version of the evaluation framework has been developed. A challenge for the project has been how to elucidate 
the key and/or unique elements of the learning design that enable the facilitation of a high quality learning 
experience for studeits. The strategy thus adopted is to request a description of the learning design by the 
designer(s) in a contextualised form in terms of the following: 

■ The learning activities (and their sequence) that students are required to do. 

■ The resources that are required to support the activities. 

■ The support mechanisms that characterise the learning design, eg., role of the instructor, establishment 
of collaborative teams, etc. 

In addition, all resources utilised by the students along with any evaluation data or findings have also been 
submitted. The evaluation review framework was implemented in two phases. The purpose, process and 
outcome for each phase are outlined in table 1 . 



Table 1: Evaluation Redevelopment Framework Implementation 


Phase 


Purpose 


Process 


Outcome 


Phase 1 


• Identify and 


Completion of two instruments: 


• Detailed description of the 




describe the 


1 . Learning Design Submission Form — to be 


learning design from the 




learning design. 


completed by the designer(s) of the learning 


designer(s). 




• Assess the data 


design. 


• Decision whether to proceed 




sources 


2. Learning Design Assessment Form — to be 


to Phase 2. (If to proceed to 
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Table 1: Evaluation Redevelopment Framework Implementation 


Phase 


Purpose 


Process 


Outcome 




provided and 
determine 
whether to 
proceed to 
Phase 2. 


completed by the Project Manager on receipt 
of the completed Learning Design 
Submission Form 


Phase 2, details about 
Evaluation Team specified.) 


Phase 2 


Evaluation of the 
learning design in 
terms of: 

• Its potential to 
facilitate high 
quality learning 
experiences for 
students. 

• Its suitability 
for 

redevelopment 
in a more 
generic form. 


Completion of one instrument; Learning 
Design Evaluation Form . 

The instrument comprises eight questions; 

• Questions 1 to 4 address the potential of the 
learning design to foster high quality 
learning. 

• Question 5 addresses how the technologies 
employed facilitate the learning design. 

• Questions 6 and 7 are designed to elucidate 
the key and/or unique elements of the 
learning design. 

• Question 8 requires a judgement to be made 
about whether the learning design is suitable 
for redevelopment in a more generic form. 

The instrument is to be completed individually 
by two evaluators. The evaluators are to reach 
consensus and submit one completed 
Evaluation Form . 


• Judgement ofthe potential of 
the learning design to foster 
high quality learning. 

• Generic description of the 
learning design. 

• Judgement ofthe learning 
design’s suitability for 
redevelopment in a more 
generic form. 



The current stage of the project 

The current stage of the project is applying the framework to a number of Learning Design exemplars. The 
outcome from this activity is intended to provide: 

1. Documentation of Learning Designs identified as having potential for redevelopment in a more 
generic form; and 

2. A formative evaluation of the framework and its operationalisation to a level of “robustness” 
deemed adequate by the project team. 

In this stage we have: 

■ Identified over 50 potential ICT -based learning exemplars for examination. 28examples will undergo 
full evaluation. Some strategies employed to compile the list of exemplars included: nominations made 
from the project team, review of past CUTSD (Committee for University Teaching and Staff 
Development) projects; and a review of relevant literature sources. 

■ Established approximately 30 Evaluation Teams. Evaluation Teams comprise pairs of national and 
international experts in the use of information and communication technologies for teaching and 
learning in Higher Education. Nominations have been made by the Project Core Team and by 
participants who attended a national flexible learning conference in July 2001. 



From the feedback received it appears that many of the learning designs the core team identified for evaluation 
are suitable for redevelopment but the expense may not warrant it. Preliminary analysis of the evaluations 
returned to date raises the following questions for the core team to consider: 

■ Do we have a large enough range of types of learning designs? 

■ Is the evaluation framework adequate to enable reviewers’ analysis of the generic attributes of any 
learning des ign? 

■ Are reviewers capable of distinguishing between the task of software evaluation and this task of 
evaluating the underpinning learning design? 

■ Are reviewers evaluating a particular implementation of a learning design for redevelopment or are they 
taking into account other implementations of that particular learning design? 

■ Do we have an adequate definition of learning design and has it been communicated to the evaluators? 

■ Is the evaluation process able to cover levels of granularity in learning designs? Should the tasks, 
resources and supports in the learning design also be evaluated? 

■ Can learning designs really be context -free? 
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■ Will the essence of a learning design always be translatable into a software product or would it instead 
be better to write guidelines for good design? 

In order to ensure the range of learning designs reviewed is not limited in representation of a broad range of 
pedagogical approaches, the project team explored the development of a Learning Design classification 
framework. An analysis of the learning design exemplars collected to produce a grounded learning design 
categorisation plus a review of categorisations of learning designs in the literature will be presented at the 
conference. The project Web site aims to inform people of the progress and provide access to the resources and 
materials as they have been developed. (http://www.Iearningdesigns.uow.edu.au) 
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